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Abstract 

A  method  is  presented  for  evaluating  the 
performance  of  a  vehicle  pose  measurement 
system  (e.g.,  GPS,  inertial  sensors,  etc.).  The 
method  supports  evaluations  of  the  system  on  a 
vehicle  moving  at  high  speeds.  An  example  is 
provided  to  illustrate  the  method. 
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1  Introduction 

Precise  knowledge  of  the  pose  of  a  moving 
ground  vehicle  is  important  in  such  applications 
as  navigation,  safety,  robotics,  metrology  and 
surveying.  Vehicle  pose  measurement  (VPM) 
system  manufacturers  typically  state  the 
accuracy  of  their  systems  under  static  conditions, 
that  is,  with  the  system  stationary  over  time. 
Latency  and  synchronization  errors  are  difficult 
to  detect  under  static  conditions.  In  this  paper,  a 
method  is  described  for  evaluating  the 
performance  of  a  VPM's  ability  to  measure  the 
position  of  a  moving  vehicle.  The  method 
requires  that  the  VPM  have  a  synchronization 
interface,  which  we  call  a  sync  pulse  interface. 
This  interface  is  often  incorporated  in  a  VPM  to 
support  photogrammetry  applications  where  a 
camera's  vertical  sync  latches  a  pose 
measurement.  In  the  method  presented  in  this 
paper,  a  photo  emitter/detector  sensor  that  senses 
energy  reflected  off  a  reflective  target  generates 
a  sync  pulse.  By  mounting  the  photo  sensor  on 
the  vehicle  and  placing  a  reflective  target  on  the 
ground,  pulses  are  generated  whenever  the 
vehicle  drives  directly  over  the  target.  Repeated 
measurements  of  the  target  as  the  vehicle  drives 
by  are  used  to  determine  the  repeatability  in  the 
VPM's  measurement  of  the  target  location. 

The  National  Institute  of  Standards  and 
Technology  (NIST)  Intelligent  Systems  Division 


is  pursuing  work  in  several  areas  of  intelligent 
vehicles,  including  research  in  autonomous 
mobility  and  development  of  performance 
measurement  techniques.  Various  testbed 
vehicles  are  used  in  support  of  several  projects, 
and  each  has  been,  or  will  be,  outfitted  with  a 
VPM  system.  These  systems  are  used  in  at  least 
two  distinctive  ways.  For  autonomous  vehicles, 
the  VPM  is  a  principal  component  of  a  real-time 
navigation  system,  providing  necessary 
information  for  the  vehicle  to  move  through  the 
environment.  A  somewhat  different  need  is  met 
by  the  ability  of  the  VPM  system  to  capture 
appropriate  information  in  real-time  that  can  be 
combined  with  other  sources  of  information  after 
the  fact  to  generate  more  precise  pose 
information  than  is  available  in  real-time.  This 
meets  a  metrology  need,  providing  improved 
accuracy  to  evaluate  performance  of  the  vehicle 
itself  and  its  various  sensors. 

This  paper  describes  a  method  to  determine  or 
confirm  the  performance  of  an  implemented 
VPM  on  a  testbed  vehicle.  Since  the  VPM 
examined  in  this  paper  is  intended  for  metrology 
applications,  steps  are  included  in  the  method  to 
post  process  the  data  to  obtain  the  highest 
possible  accuracy.  The  method  described  could 
also  be  used  to  obtain  real-time  performance 
information.  This  is  accomplished  by  analyzing 
the  real-time  solutions  as  opposed  to  the  post- 
processed  solutions.  The  method  consists  of  a 
way  to  precisely  trigger  vehicle  position 
measurements,  a  procedure  for  collecting  data 
and  a  way  to  analyze  the  repeatability  in  the 
vehicle  position  measurements.  Although  VPMs 
are  capable  of  providing  a  full  position  and 
orientation  solution,  this  current  effort  addresses 
only  the  position  measurement  capability.  In 
the  following  sections,  the  method  is  described 
in  the  context  of  evaluating  a  specific  VPM, 
though  the  described  method  is  applicable  to  any 
VPM  that  supports  a  trigger  mechanism. 
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Figure  1  Note  the  downward  looking  photo  sensor,  the  retroreflector  on  the  ground  and  the  vertical 
rod  to  guide  the  driver. 


2  Test  Procedures 

The  vehicle  used  in  the  evaluation  is  a  full  size 
passenger  sedan.  The  installed  VPM  is  a  system 
that  integrates  a  dual  frequency  carrier  phase 
Global  Positioning  System  (GPS),  a  secondary 
GPS,  an  inertial  measurement  unit  (IMU) 
consisting  of  accelerometers  and  fiber  optic 
gyros,  a  wheel  encoder,  and  a  system  control  and 
data  collection  computer.  This  system  is 
configured  to  capture  and  provide  solutions  at  a 
200  Hz  data  rate.  In  addition,  it  is  configured  to 
capture  all  data  required  for  later  post 
processing. 

The  system  is  augmented  with  an  external  photo 
sensor  connected  to  the  sync  pulse  interface. 
When  triggered,  the  result  is  the  notation  in  a  file 
of  an  event  occurrence,  along  with  the  time  of 
the  event  with  microsecond  time  precision. 

Since  triggers  will  be  caused  during  vehicle 
motion,  latency  in  the  assertion  of  this  signal  will 
be  reflected  in  position  errors  in  the  resulting 
data.  For  this  reason,  a  low  latency  photo  sensor 
(300  ms)  was  employed.  During  post 
processing,  pose  solutions  are  determined  for 
selected  events  of  interest.  The  sensor  was 
mounted  slightly  in  front  of  the  vehicle  bumper, 
in  line  with  the  vehicle  driver  position.  A 
vertical  rod  above  the  sensor  assists  the  driver  in 
steering  the  sensor  over  a  target  while  driving. 
The  target  is  a  simple  retroreflective  surface, 
constructed  by  affixing  a  layer  of  retroreflector 
material  (available  in  sheets)  to  an  aluminum 
disk.  A  disk  of  15.24  cm  (6  in)  diameter  is  used. 
This  size  was  selected  as  large  enough  to  enable 


the  driver  to  successfully  steer  the  vehicle  over  it 
at  speeds  of  interest  most  of  the  time,  even  in  a 
grassy  and  somewhat  bumpy  field,  but  small 
enough  to  keep  the  target  detection  points  close 
to  a  surveyed  point.  In  this  test  a  National 
Geodetic  Survey  (NGS)  survey  marker,  flush 
with  the  ground,  is  used.  The  retroreflector  is 
simply  centered  on  top  of  it. 

The  configuration  of  the  photo  sensor,  its 
mounting,  and  target  is  shown  in  Figure  1 . 

To  evaluate  the  VPM's  measurement 
performance,  the  vehicle  is  driven  over  the  target 
from  a  variety  of  directions  while  collecting  all 
data  necessary  to  compute,  via  post  processing, 
the  position  of  the  sensor  at  the  time  of  target 
crossing.  In  the  analysis  phase,  the  repeatability 
of  the  measurements  of  the  target's  radius  and  the 
location  of  the  target  with  respect  to  the  survey 
marker  produce  a  measure  of  performance  of  the 
VPM  system. 

The  position  solutions  are  computed  for  the 
location  of  the  sensor  (actually  for  a  point  on  the 
ground  directly  below  the  sensor).  This  requires 
a  transformation  of  coordinate  frames  between 
the  VPM  system  and  the  position  of  interest 
below  the  sensor.  This  transformation  depends 
on  knowledge  of  the  translations  and  rotations 
between  the  two  reference  frames.  In  the  case  of 
this  implementation,  translation  and  rotation 
measurements  between  the  IMU  reference  frame 
and  the  sensor  reference  frame  are  needed. 

These  measurements  are  entered  into  the  VPM 
system.  Determination  of  the  translation  and 
rotation  parameters  is  performed  in  two  steps. 


The  first  is  a  best-effort  measurement  of  the  (x, 
y,  z)  distances  between  the  IMU  and  the  point 
below  the  sensor.  Performing  these 
measurements  is  physically  awkward  because  of 
the  location  of  the  IMU  in  the  vehicle  trunk  and 
the  sensor  location  at  the  front  bumper.  An 
attempt  was  made  to  mount  the  IMU  to  align 
with  the  major  vehicle  axes  as  well. 

Since  these  measurements  are  difficult  and 
somewhat  prone  to  error,  data  is  also  collected 
during  the  test  run  which  is  used  to  calibrate  the 
system  by  adjusting  the  translation  and  rotation 
parameters  as  required.  Several  events  are 
triggered  by  driving  the  vehicle  sensor  over  the 
target  as  slowly  ("creep")  as  possible  and  as  well 
centered  laterally  as  possible.  These  creep 
events  are  collected  for  approaches  to  the  target 
from  four  directions.  The  slow  speed  is  intended 
to  eliminate  any  significant  data  latency  errors. 
Positions  of  events  collected  in  this  manner 
should  conform  to  the  edge  of  the  target,  and  the 
calibration  parameters  are  adjusted  slightly  to 
yield  solutions  consistent  with  the  known  target 
diameter.  This  “creep”  data  also  helps  detect  any 
movement  of  the  sensor  that  may  have  occurred 
between  tests. 

After  collecting  the  initial  creep  calibration  data, 
the  vehicle  was  driven  over  the  target  at  a 
number  of  speeds  and  from  a  number  of 
directions.  Speeds  were  limited  to  about  10.6 
m/s  (=20  mph)  due  to  the  roughness  of  the  grassy 
field  where  these  tests  were  performed.  (The 
sensor/retroreflector  system  has  been  tested 
successfully  at  highway  speeds  on  a  roadway  as 
well.)  Approximately  40  events  and  their 
associated  data  were  collected  in  this  way, 
including  a  second  set  of  creep  points  after 
completion  of  the  vehicle  at-speed  runs. 

3  Post  Processing 

Since  the  purpose  of  this  test  was  to  determine 
the  maximum  accuracy  possible  from  the  VPM, 
post  processing  was  conducted  on  the  data.  If 
the  purpose  of  a  test  is  to  evaluate  the  real-time 
performance  of  a  VPM,  then  this  step  would  not 
be  performed. 

The  data  logged  on  the  vehicle  during  the  test  is 
retrieved,  and  the  real-time  navigation  solution, 
though  not  the  subject  of  this  paper,  is  examined 
to  confirm  the  existence  of  good  data  depicting 
an  appropriate  vehicle  trajectory.  The  raw  data 


is  then  post  processed  to  obtain  the  more  precise 
solution  for  the  events  of  interest. 

To  enhance  the  quality  of  the  solution, 
differential  GPS  post  processing  is  used.  A 
detailed  explanation  of  this  processing  is  beyond 
the  scope  of  this  paper,  but  the  approach 
essentially  makes  use  of  information  collected 
from  another  nearby  GPS  receiver  (base  station) 
at  an  accurately  known  location  to  remove 
(during  post  processing)  certain  kinds  of  errors 
from  the  reported  position  of  the  rover  (our 
vehicle). 

For  these  tests,  we  used  a  National  Geodetic 
Survey  (NGS)  Continuously  Operating 
Reference  Station  (CORS)  located  in 
Gaithersburg,  MD  known  as  GAIT. 

The  NGS,  an  office  of  NOAA's  National  Ocean 
Service,  coordinates  a  network  of  continuously 
operating  reference  stations  (CORS)  that  provide 
GPS  carrier  phase  and  code  range  measurements 
in  support  of  3 -dimensional  positioning  activities 
throughout  the  United  States  and  its  territories. 
(See  http://www.ngs  .noaa.  go v/CORS/) 

The  reference  station  data  for  station  GAIT  is 
downloaded  from  the  NGS  for  the  appropriate 
period  during  which  we  performed  the  test,  and 
used  by  post  processing  software  to  enhance  the 
GPS  solution.  Further  post  processing  is 
performed  to  integrate  this  GPS  solution  with  the 
raw  data  from  the  VPM  system  sensors  (IMU 
and  wheel  encoder).  Smoothing  algorithms  are 
executed,  and  interpolation  of  navigation 
solutions  for  the  recorded  events  is  performed. 
The  result  is  a  file  with  a  full  navigation  solution 
for  each  event.  The  solution  information  in  this 
file  is  analyzed  below. 

4  Analysis 

Two  types  of  VPM  measurement  errors  are 
estimated:  target  (i.e.,  reflector)  location  and 
target  radius.  The  estimated  location  error  is 
derived  as  the  difference  between  the  surveyed 
(static)  location  of  the  target  and  VPM  measured 
(dynamic)  location  of  the  target.  The  estimated 
radius  error  is  the  difference  between  the  known 
radius  of  the  target  and  VPM  measured  radius  of 
the  target.  The  location  error  may  be  left 
uncalculated  if  survey  data  is  not  available.  The 
following  process  is  used  to  compute 
uncertainties. 


1.  Put  event  points  in  a  convenient  local 
coordinate  system.  First,  the  event  points  (VPM- 
measured  coordinates  of  target's  edge)  are 
transformed  from  latitude  and  longitude 
coordinates  into  UTM  coordinates  (see 
velvet.tec.armv.mil/access/milgov/fact  sheet/ geo 

trans.html)  so  that  errors  may  be  expressed  in 
meters.  Second,  if  available,  the  known  location 
of  the  target  is  subtracted  from  each  event  point. 
This  places  the  event  points  in  a  coordinate 
system  whose  origin  is  the  surveyed  location  of 
the  target. 


Figure  2.  Example  measurement  points 
surrounding  the  target's  edge. 

2.  Determine  the  target  location  error.  A  circle 
is  fit  to  the  event  points.  The  fit  produces  an 
offset  vector,  O,  indicating  the  center  of  the 
circle.  Figure  2  shows  an  example  set  of  data 
points  with  the  offset  vector  to  the  center  of  the 
circle.  If  the  origin  of  the  coordinate  system  is 
the  surveyed  location  of  the  target,  then  the 
magnitude  of  O,  e/,  describes  the  estimated  target 
location  error. 

3.  Determine  the  target  radius  error  and 
uncertainty.  First,  the  offset  vector  of  the  circle 
is  subtracted  from  each  event  point.  This  places 
the  event  points  into  a  coordinate  system  with 
the  center  of  the  circle  at  the  origin  (see  Figure 
3).  Then  the  distance  of  each  event  point  from 
the  origin,  d,,  is  computed.  These  distances  are 
the  measured  radii  of  the  target.  The  mean  and 
standard  deviation  of  the  radius  measurements, 

[ir  and  Sr,  are  computed.  The  estimated  target 
radius  error,  ei,  is  the  difference  between  the 
mean  of  the  target  radius  measurements  and  the 


target  radius  measured  by  hand.  Choosing  a  95 
%  level  of  confidence,  the  component  of  the 
expanded  uncertainty  due  to  data  scatter  in  the 
radius  measurement  is: 

(1) 

Where  k  =  2  for  N>30  and  k  equal  to  a  t-factor 
obtained  from  a  t-distribution  for  N  <  30. 


Figure  3.  Measurements  points 
translated  such  that  center  of  circle  is  at 
origin.  The  distance  of  each  point  is  the 
measured  radius  of  the  target 


Figure  4  shows  a  plot  of  the  event  points 
collected  when  the  vehicle  traveled  over  a 
circular  target  with  a  radius  of  7.62  cm  (3  in). 

The  origin  of  the  plot  coincides  with  the  survey 
marker's  coordinates,  in  this  case,  the  coordinates 
of  the  NGS  survey  marker  where  the  target  was 
placed.  The  large  circle  is  the  target  drawn  to 
scale.  Each  event  point  is  plotted  as  a  small 
circle  with  an  attached  velocity  vector  (origin  of 
vector  is  inside  of  small  circle  and  points  away 
from  circle  in  direction  of  travel).  The  vectors 
are  scaled  to  fit  to  the  plot.  The  largest  velocity 
vector  is  labeled  10.6  m/s  to  provide  a  scale  for 
comparison  (label  is  at  end  of  vector  opposite 
small  circle  shown  on  left  side  of  plot).  The 
results  of  the  analysis  of  these  measurements 
indicate  a  location  error  of  0.3  cm,  an  estimated 
radius  error  of  0.2  cm  and  a  radius  uncertainty  of 
1.3  cm  (95  %  level  of  confidence). 


Figure  4  Plot  of  event  points  (shown  as  small  circles)  triggered  when  vehicle  is  driven  over  the 
edge  of  a  circular  target  (large  circle)  with  a  radius  of  7.62  cm.  Vectors  at  each  event  point  are 
vehicle  velocity  at  event  point.  The  largest  velocity  was  10.6  m/s  (left  side  of  target). 


5  Conclusions 

Further  work  is  needed  in  several  areas.  This 
effort,  an  initial  step  in  developing  a 
performance  measurement  capability,  focused 
only  on  vehicle  position  information  in  x  and  y. 
Height  (z)  data  was  collected  but  the  data  has  not 
yet  been  analyzed.  The  testing  methodology 
needs  to  be  expanded  to  include  orientation  (roll, 
pitch,  yaw)  capabilities  of  VPM  systems  as  well. 
That  information,  along  with  position 
information,  is  critical  in  registering  the  data 
received  from  intelligent  vehicle  sensors  such  as 
cameras  and  laser  scanners,  and  directly  affects 


the  correctness  of  the  model  of  the  world  being 
maintained  by  the  vehicle  computers.  In 
addition,  the  scope  of  the  work  here  included 
differential  post  processing  of  GPS  data  and 
further  post  processing  of  all  the  navigation 
sensor  data.  This  is  appropriate  for  metrology 
purposes.  For  navigation  of  intelligent  vehicles, 
the  real-time  solution  is  important,  and  methods 
for  determining  the  quality  of  the  real-time 
navigation  solution  should  be  explored.  Further, 
a  real-time  navigation  solution  may  make  use  of 
real-time  differential  GPS  corrections  received 
by  the  vehicle  during  operation.  The  method 
described  here  can  be  used  to  examine  position 
determination  performance  for  those  types  of 


systems  as  well,  and  should  be  extended  to 
characterize  orientation  measurement 
performance.  Further,  these  tests  were 
conducted  with  good  GPS  satellite  coverage. 
While  gaps  in  satellite  coverage  occurred,  a  test 
specifically  excluding  satellite  info  for 
prescribed  periods  would  enable  performance 


analysis  of  the  VPM  systems  when  they  rely 
more  on  their  inertial  components. 
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